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[57] ABSTRACT 

This is a distributed computing system providing flexi- 
ble fault tolerance; ease of software design and concur- 
rency specification; and dynamic balance of the loads. 
The system comprises a plurality of computers each 
having a first input/output interface and a second input- 
/output interface for interfacing to communications 
networks each second input/output interface including 
a bypass for bypassing the associated computer. A 
global communications network interconnects the first 
input/output interfaces for providing each computer 
the ability to broadcast messages simultaneously to the 
remainder of the computers. A meshwork communica- 
tions network interconnects the second input/output 
interfaces providing each computer with the ability to 
establish a communications link with another of the 
computers bypassing the remainder of computers. Each 
computer is controlled by a resident copy of a common 
operating system. Communications between respective 
ones of computers is by means of split tokens each hav- 
ing a moving first portion which is sent from computer 
to computer and a resident second portion which is 
disposed in the memory of at least one of computer and 
wherein the location of the second portion is part of the 
first portion. The split tokens represent both functions 
to be executed by the computers and data to be em- 
ployed in the execution of the functions. The first input- 
/output interfaces each include logic for detecting a 
collision between messages and for terminating the 
broadcasting of a message whereby collisions between 
messages are detected and avoided. 

33 Claims, 5 Drawing Sheets 
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DISTRIBUTED COMPUTING SYSTEM WITH 

DUAL INDEPENDENT COMMUNICATIONS 
PATHS BETWEEN COMPUTERS AND 
EMPLOYING SPLIT TOKENS 

ORIGIN OF THE INVENTION 

The invention described herein was made in the per- 
formance of work under a NASA contract and is sub- 
ject to the provisions of Public Law 96-517 (35 USC 
202) in which the Contractor has elected not to retain 
title. 

TECHNICAL FIELD 

A distributed computing system comprising a plural- 
ity of interconnected computers operating in a novel 
distributed variation of dataflow execution mode to 
provide flexible fault tolerance, to ease software design 
and concurrency specification, and to dynamically bal- 
ance the loads; a distributed operating system resident 
on all computers; a real-time multiple-access global bus 
using Lanning code arbitration with deterministic ac- 
cess delay; and a point-to-point circuit-switched net- 
work. 

DESCRIPTION OF THE PRIOR ART 

Future spacecraft computers will require greater 
autonomy, flexibility, fault tolerance and higher 
throughput. The necessary solution is a new flexible 
computational system that will support a wide range of 
instruments, attain a tolerable arbitrary level of fault 
and damage tolerance, and provide a greater range of 
throughput capability. Designers of future spacecraft 
systems will confront a variety of new applications in 
onboard computing. The traditional roles of spacecraft 
computers are expanding to encompass tasks previously 
assigned to ground based systems. Increased autonomy 
is becoming a major goal at all levels to improve both 
efficiency and reliability. Greater flexibility will be 
needed to accommodate evolving requirements for 
existing spacebome systems. Also, new instruments and 
engineering subsystems will require greater computa- 
tional throughput. 

Considering the range of payloads on such diverse 
systems as the Space Station, Earth Observing System, 
planetary spacecraft, and a host of Space Shuttle experi- 
ments, there exists the potential need for many hundreds 
of general purpose computing systems. The Space Sta- 
tion alone may require dozens, if not hundreds, of com- 
puters when payloads are included. 

Reliability issues include the operating environment 
as well as fault and damage tolerance. One environmen- 
tal issue that has plagued current generation computing 
components is their intolerance to ionizing radiation 
damage and to a single event upset (SEU). SEUs arise in 
small feature size parts due to trapped ions, solar flares 
and cosmic rays. The ability for space systems to evolve 
to a newer, high density technology such as VHSIC is 
a daunting problem. Solutions, in many cases, may have 
to come from architectural restructuring in new and 
novel ways. 

Computer system architectures have various facets. 
There is the structure of the computers themselves, i.e. 
one or many. Operating systems and the associated 
support software can be configured in many ways. Then 
there is the consideration of how multiple computers 
will communicate with one another. Early systems 
employed one large computer doing the work on a 
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batch input basis. Later, data acquisition and processing 
functions were made “real time” in many cases but still 
employing one large computer to do the work. Some- 
times, priority levels were assigned to the various tasks 
5 within the computer and the tasks shared the computing 
ability of the hardware on an unequal basis with the 
more important tasks being done first and the tasks of 
lesser importance being done on a time available basis. 
The advent of multi-processors (i.e. multiple computers 
1° linked together to perform a common function) created 
many new problems for the computer architects. Partic- 
ularly sensitive in the military and space environments 
are the concepts of fault tolerance and “graceful degra- 
dation”; that is, with one computer, when it is not work- 
15 ing, nothing gets done, but, with multiple computers, if 
a portion of the computing capability is inoperative for 
any reason, it is expected that the remaining capability 
will take over at least critical functions and keep them 
operational. 

A few examples of various architectural approaches 
known in the computer art are depicted in simplified 
form in FIGS. 1-4. FIG. 1 depicts a contemporary 
token ring network 10 wherein a plurality of computer 
25 nodes 12 are disposed along a circular communications 
path 14. To send messages between one another, each 
node samples the data on the path 14 looking for a 
“token” (i.e. a particular bit pattern not otherwise ap- 
pearing) signifying the end of a message. When the 
30 token is located, the node 12 puts its message on the 
path 14 immediately following the found token and 
appends a new token at the end of its message to flag the 
end thereof. If a token is not found within a given time, 
the node 14 puts its message on anyway assuming net- 
35 work failure of some sort and hopes for the best. In such 
case, the network 10 degrades into what is referred to as 
a “contention” network wherein each node simply puts 
its messages on the network at will and if they are not 
acknowledged by the receiver as having been properly 
4 q received within a given time, repeats the process until 
successful sending and receiving has been achieved. 
Some inter-computer communications networks oper- 
ate on that basis all the time. Obviously, in either the 
token ring approach or the contention approach, some 
45 nodes can dominate the communications path and pre- 
vent other nodes from communicating at all. All in all, 
such approaches which may be acceptable in some 
ground based general computer systems are not accept- 
able for systems employed in the space applications 
50 being considered herein. 

A common prior art approach to both hardware and 
software interconnection and hierarchy is depicted in 
FIG. 2. In this case, a plurality of task-oriented, work- 
ing level computers 16 are interconnected in a multi- 
55 processor environment. The software in the computers 
16 is primarily that required to perform user functions 
or tasks such as data acquisition and processing. Super- 
vising computers 18 are connected to monitor the com- 
puters 16 and contain operating system type software 
60 for assigning and reassigning tasks to the computers 16 
for the purpose of distributing the workload for maxi- 
mum system efficiency and to accomplish fault toler- 
ance and graceful degradation as necessary. Typically, 
there is a master computer 20 which has ultimate con- 
65 trol of the whole system. The master computer 20 pro- 
vides the system interface and is used to start up and 
configure the system in general, among other tasks. As 
can be appreciated, such an approach is complex to 
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create, complex to debug, and subject to high overhead 
problems. Moreover, there must be redundancy on the 
supervising and master computer levels to prevent sys- 
tem failure if one of those computers fails or is de- 
stroyed. 

FIG. 3 depicts a data-driven approach to computer 
architecture which is known in the art. Each task 22 is 
caused to execute when the data from the preceding 
tasks 22 required for its execution are provided by those 
preceding tasks. 

Finally, FIG. 4 depicts a basic eight node element of 
a so-called “hypercube” approach to computer archi- 
tecture wherein as many as 64,000 individual computing 
nodes 16 are interconnected. The hypercube is charac- 
terized by implementing an approach such as that de- 
picted in FIG. 2. The nodes 16 are interconnected by a 
first communications network 24 over which the nodes 
16 transfer their data, etc. The supervising computers 18 
are connected to the nodes 16 by a second communica- 
tions network 26 over which reconfiguration and redis- 
tribution/reassignment of task instructions are sent so 
that the same communications paths are not employed 
for the two different purposes. 

Many fault tolerance approaches with prior art com- 
puter architectures are hardware intensive, requiring 
elaborate hardware synchronization mechanisms. Even 
with this investment in hardware, the application soft- 
ware is often subject to constraints, and requires addi- 
tional overhead to support the underlying architecture. 
The degree of fault tolerance achieved in these designs 
is dictated by the architecture, not by the needs of the 
software designer. Furthermore, damage tolerance is 
not currently achievable by the same mechanisms 
which provide fault tolerance, without significant per- 
formance penalties. 

Finally, there is the issue of software cost and reliabil- 
ity. Very large real-time programs for uniprocessors are 
difficult enough to write and test. Distributed comput- 
ing systems aggravate the problem by adding topologi- 
cal considerations. To be fully effective, future multi- 
processing systems must consider software develop- 
ment issues. 

What is required is a single architectural concept to 
deal with these requirements. The intent is to provide 
the spacecraft system designer with a system capable of 
handling a general class of computing applications, e.g., 
payload controllers, smart sensors, robotics, engineer- 
ing subsystems and moderate throughput signal pro- 
cessing. The required capability is not intended to sup- 
plant special purpose hardware such as high throughput 
signal processors, or incircuit microprocessors. 

There is presently no flexible design which will sup- 
port arbitrary levels of fault and damage tolerance and 
provide a range of throughput capability through mod- 
ularity and parallelism. There is also no present soft- 
ware environment which deals cleanly with the issues 
of concurrency, productivity, reliability, and fault toler- 
ance while permitting the generation of applications 
software using commercially available tools. 

DISCLOSURE OF THE INVENTION 

The present invention has achieved its desired objec- 
tives through use of modularity and parallel computa- 
tion, based on a multiprocessor system interconnected 
by a global bus and an independent data network, em- 
ploying a distributed operating system operating in 
dataflow mode of task execution. A multiprocessor 
architecture was chosen since it offers easy expandabil- 
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ity and also provides the necessary basis for a fault-tol- 
erant and gracefully degradable system. More specifi- 
cally, the present invention is a multicomputer system 
consisting of loosely coupled processor/memory pairs 
connected by global and point-to-point communication 
systems. This approach is preferable to a shared mem- 
ory implementation which would require hardware 
intensive approaches to fault tolerance. 

More particularly, this invention is a computer sys- 
tem comprising, a plurality of computers, each com- 
puter having first input/output interface means for in- 
terfacing to a communications network and second 
input/output interface means for interfacing to a com- 
munications network, each second input/output inter- 
face means including bypass means for bypassing the 
associated computer with a bit stream passing through 
the second input/output interface means; global com- 
munications network means interconnecting respective 
ones of the first input/output interface means for pro- 
viding respective ones of the computers with the ability 
to broadcast messages simultaneously to the remainder 
of the computers; and, meshwork communications net- 
work means interconnecting respective ones of the 
second input/output interface means for providing re- 
spective ones of the computers with the ability to estab- 
lish a communications link with another of the comput- 
ers through the bypass means of the remainder of the 
computers. 

The preferred embodiment includes several addi- 
tional features to accomplish the overall objectives 
thereof. Specifically, each computer is controlled by a 
resident copy of a common operating system. Commu- 
nications between respective ones of the computers is 
by means of split tokens each having a moving first 
portion which is transmitted from computer to com- 
puter and a resident second portion which is disposed in 
the memory of at least one of computers and wherein 
the location of the second portion is part of the first 
portion. As a major point of novelty, the split tokens 
represent both functions to be executed by computers 
and data to be employed in the execution of the func- 
tions. 

Additionally, the first input/output interface means 
each includes means for detecting a collision between 
messages being broadcast over the global communica- 
tions network means and for terminating the broadcast- 
ing of a message in favor of a message with superior 
right to use of the global communications network 
means whereby collisions between messages from a 
plurality of computers are detected and avoided. 

Additionally, the bypass means of the second input- 
/output interface means at each computer includes a 
receiver for receiving inputs to the associated com- 
puter, a transmitter for transmitting outputs from the 
associated computer, a plurality of inputs and outputs 
connected to respective ones of the inputs and outputs 
of others of computers interconnected by the meshwork 
communications network means, and crossbar switch 
means interconnecting the inputs and the outputs for 
bypassing the receiver and the transmitter at the associ- 
ated computer and for interconnecting one of the plu- 
rality of inputs to one of the plurality of outputs 
whereby a bypass link between the ones of the plurality 
of inputs and outputs is established; and, the bypass 
means of the second input/output interface means at 
each computer includes means for sensing a unique 
signal on the meshwork communications network 
means indicating that the bypass link is to be terminated 
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and for causing the crossbar switch means to open the 
bypass link when the unique signal is sensed. 

DETAILED DESCRIPTION OF THE 
DRAWINGS 

FIG. 1 is a simplified drawing representing a prior art 
token passing ring computer network. 

FIG. 2 is a simplified drawing representing a prior art 
computer network in which multiple levels of software 
system control are implemented. 

FIG. 3 is a simplified drawing representing a prior art 
approach of driving a system by data tokens. 

FIG. 4 is a simplified drawing representing a socalled 
hypercube in which multiple computers are intercon- 
nected with a data network and connected to a supervi- 
sory computer via a separate control message network. 

FIG. 5 is a simplified block diagram drawing of the 
computer system of the present invention showing the 
operating system duplicated in each of the computer 
modules, the modules connected to a global bus for 
intermodule communication, and the modules loosely 
interconnected by a meshwork providing module-to- 
module linking for the transfer of data and functions. 

FIG. 6 is a simplified block diagram of a representa- 
tive module in the present invention showing the con- 
nection to the global bus, the connection to the mesh- 
work providing module bypass for intermediate mod- 
ules, the providing of global time via the global bus, and 
the common operating system resident in each of the 
modules. 

FIG. 7 is a drawing showing how the function/data 
tokens employed in the present invention are split into 
moving and resident portions. 

FIG. 8 is a drawing of the structure of a typical mes- 
sage as broadcast over the global bus in the present 
invention. 

FIG. 9 is a drawing showing the intermessage struc- 
ture of message according to FIG. 8 when there is a 
contention between modules in progress for the global 
bus. 

FIG. 10 is a drawing showing the intermessage struc- 
ture of message according to FIG. 8 when there is not 
a contention between modules for the global bus. 

FIG. 11 is a block diagram of the meshwork inter- 
face. 

FIG. 12 is a block diagram of the meshwork control- 
ler. 

FIG. 13 is a drawing showing the effect of intercon- 
necting wires between the transmitters and receivers in 
narrowing the data pulses and changing their orienta- 
tion to the clock. FIG. 14 is a circuit diagram of the 
repeaters employed in the present invention to eliminate 
the problem shown in FIG. 13. 

FIG. 15 is a block diagram showing how meshwork 
controllers can be interconnected to increase their ca- 
pacity according to the present invention. 

DETAILED DESCRIPTION OF THE 
INVENTION 

The main difficulties which arise in implementations 
for real-time applications on interconnected multiple 
computer systems are the distribution of event data in a 
timely fashion as well as software synchronization and 
scheduling. Other issues include an efficient intercon- 
necting communication system, methods for distribu- 
tion of control, task migration, support for fault toler- 
ance, and input/output handling. The present invention 
employs a concurrency technique which offers solu- 
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tions to most of these problems in the form of low reso- 
lution dataflow. 

Dataflow is central to the architecture of the present 
invention. The order in which various functions are 
performed and the means by which data and functions 
are associated is fundamental to a computer’s architec- 
ture. In a von Neuman architecture, instruction execu- 
tion is directed by an instruction pointer which passes 
control from one function to the next. There is no inher- 
ent concurrency. Data is identified through its location 
in memory. 

In a dataflow architecture, the execution of each 
function is enabled by the presence of the data required 
by that function. Data is encapsulated in tokens which 
identify the functions to which the data is destined. The 
use of the token by a function consumes the token. 
Functions produce results in the form of new tokens. 
Tokens are created, used (read) and then destroyed. 
There is no correspondence between a data item and its 
location in memory because of the encapsulation of data 
within tokens. Dataflow functions are thus constrained 
to be functions with no “side effects”. 

In the present invention, a particular function be- 
comes ready for execution (fireable) as soon as it has 
received all of the necessary input tokens. Therefore, 
several functions may be fireable at any moment. If the 
functions are distributed over the processing units of the 
system, all fireable functions are executed at the earliest 
opportunity. Hence, the maximum concurrency poten- 
tial in the algorithm is extracted. 

Dataflow systems can be categorized by the granular- 
ity of their data and functions. In a high resolution 
dataflow system, data items are quantities such as num- 
bers and booleans. Correspondingly, functions consist 
of arithmetic and logical operations, simple decisions, 
and so on. A high resolution dataflow program consists 
of a very large number of functions requiring a large 
number of special purpose processors to extract all of 
the potential concurrency of a program. 

Lower resolution dataflow implies increased com- 
plexity of both data structures and functions. In a low 
resolution system as implemented by the present inven- 
tion, data might consist of lists, arrays, and other such 
data structures. Similarly, functions could be complex 
routines such as sorting, and matrix inversion. Low 
resolution dataflow extracts less potential concurrency 
from an algorithm, but it does this with less scheduling 
and synchronization overhead. This is an important 
consideration in the achieving of the overall objectives 
of the present invention. 

Dataflow programs can be represented in a number 
of ways. Languages such as LUCID are similar in ap- 
pearance to conventional high-level languages and are 
designed to ease the translation of source code into 
dataflow primitives. A dataflow program can also be 
expressed as a directed graph in which each node repre- 
sents a function, and connecting arcs define the paths of 
tokens from source to destination. This is substantially 
an implementation of the representation of FIG. 3. 

Many features of low resolution dataflow are appeal- 
ing from an architectural point of view. For example, 
there is the ease of concurrency expression. In a data- 
flow graph, concurrency potential exists both explicitly 
in the form of parallel paths which may be executed 
simultaneously, as well as implicitly through pipelining 
in which the processing of consecutive inputs may over- 
lap. Dataflow gracefully handles transient loadings 
which might arise from asynchronous inputs or variable 
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computing times. Tagged dataflow systems provide 
automatic queuing of all data without the necessity of 
providing a separate queue for each input to a function. 
The absence of side effects and a one-to-one relationship 
of inputs to outputs makes programs easier to integrate 5 
and debug. This is a proven method of increasing soft- 
ware reliability. A graph may be verified with little 
regard to the algorithms which implement its functions. 
Any problems traced to the improper execution of a 
function can be debugged solely at the function level, 10 
and off-line. There is no sense of history in a function 
and all functions are operationally isolated form each 
other. The requirements on a function may be specified 
with no reference to other elements of the program, 
except for adherence to the structure of the tokens 15 
passed from one function to the next. Dataflow func- 
tions may be tested in isolation and changed without 
regard to the remainder of the program as long as the 
data structure conformity is preserved. 

With low resolution dataflow, consecutive inputs to a 20 
function may be processed in separate processing units 
without re-entrancy concerns. TTiere is no implied con- 
text to save or transfer. If any “memory” is required 
from one function activation to the next, the context 
must be explicitly carried by loops in the dataflow 25 
graph. Having thus described the basic attributes of the 
approach implemented by the present invention, the 
specifics of it will now be addressed in greater detail. 

As depicted in the simplified drawing of FIG. 5 and 
generally indicated as 28 therein, the computer system 30 
of the present invention consists of clusters of computer 
modules 30 interconnected by two different communi- 
cation mechanisms, the global bus 32 and the meshwork 
34. 

Each module 30 is a self-contained computer with 35 
ports connecting it to the meshwork 34 and the global 
bus 32 as depicted in FIG. 6. The global bus 32 and 
meshwork 34 as employed in the present invention are 
novel in their own regard and will be considered in 
detail later herein. Each module 30 operates autono- 40 
mously. This is a major point of novelty of the present 
invention. Additionally, the operating system is repli- 
cated on all the modules, resulting in a fully distributed 
system — which is also a major point of novelty of the 
present invention over the prior art. A module 30 also 45 
contains timers, interrupt controllers, and the capability 
for general purpose memory mapped I/O via an exten- 
sion bus as known in the art. By attaching special pur- 
pose boards to the extension bus, the capabilities of a 
module 30 can be enhanced in the usual way. For exam- 50 
pie, a mass storage device such as a digital tape recorder 
with limited or no computing capability could be di- 
rectly connected to a module 30. The system can also 
support a non-homogeneous node configuration where 
individual nodes have unique capabilities such as I/O, 55 
storage, or computational capability. Another module 
enhancement might be an array processing unit, where 
a module 32 would serve as a front end, having the role 
of exchanging input and output tokens over the mesh- 
work 34. This type of capability enhancement and the 60 
use of so-called “servers”, and the like, in a networked 
system are well known in the art and, per se, form no 
point of novelty of the present invention 

The global bus 32 is a broadcast bus used for operat- 
ing system and software synchronization functions. It 65 
should be noted at this point that “software synchroni- 
zation” refers to coordination of activities for example, 
not starting one activity until a previous activity upon 
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which it depends is finished. It is not the same as hard- 
ware synchronization involving clock signals (the 
global bus, for example, uses an asynchronous proto- 
col), nor is it the same as synchronization of the global 
bus times which involve a combination of software and 
hardware mechanisms. A cluster of modules 30 share a 
common global bus 32. The meshwork 34 is a circuit- 
switched point-to-point communications network, used 
to transport data between modules 30. Every module 30 
is connected to a number of other modules 30 in the 
system, in a topology that best suits a particular applica- 
tion; that is, the meshwork 34 does not require a “regu- 
lar” pattern of interconnection as in the case of the 
typical prior art inter-computer communications net- 
work such as those described with respect to FIGS. 1, 
2 and 4. Rather, modules 30 having a high degree of 
potential inter-module communication can be directly 
connected while modules 30 having little or no inter- 
module communication traffic can be connect so as to 
require connection through a more circuitous route. 
The meshwork 34 is also used to communicate between 
clusters of modules 30. It should be noted with particu- 
larity that the present invention incorporates two dis- 
tinct communication pathways. The global bus 32 is a 
carrier sense multiple access serial broadcast bus with 
collision detection/avoidance and equal access capabil- 
ity on a priority basis to prevent module lockout from 
access to the bus. The meshwork 34 is a point-to-point 
circuit-switched communications network with module 
bypass capability. 

THE GLOBAL BUS 

The global bus 32 is used for token announcement 
and synchronization. It is characterized by its broadcast 
capabilities, and transmission integrity mechanisms. As 
depicted in simplified form in FIG. 6, the interface at 
each module 30 to the global bus 32 employs read cir- 
cuitry 36 and transmit circuitry 38. There is also a 
global bus time 40 available which is synchronized 
across the bus 32 so that each module 30 can get a com- 
mon bus time for time tagging, and such. There may be 
more than one global bus 32 per module 30 as when 
multiple clusters are interconnected; however, for pur- 
poses of simplicity in the discussion herein, only one bus 
32 is shown. Each time a node in a dataflow graph 
completes its assigned task, it broadcasts to all the par- 
ticipating modules 30 over the global bus 32 that is has 
completed the task, as well as the identification of the 
tokens that it produced The actual data, encapsulated in 
one or more tokens, remains resident in the module 30 
where it was produced A module that requires the use 
of one or more of the tokens just produced will request 
and acquire them over the much higher bandwidth 
meshwork 34. This unique approach of the present 
invention is depicted in simplified form in FIG. 7 and is 
worthy of note by a brief digression at this point. 

As depicted in FIG. 7, the concept of a “token” in the 
present invention is different from the traditional ap- 
proaches to a token such as, for example, that described 
above with respect to prior art token passing ring com- 
munications networks. In the present invention, the 
token is “split”; that is, a token 42 comprises a moving 
portion 44 and a resident portion 46. As symbolically 
represented in the figure, the token 42 can be thought of 
as a whole which has been split into two different sized 
portions. This approach to a token is substantially the 
same for function tokens as well as data tokens. It 
should be noted in passing at this point that while data 
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tokens are known in the art, the concept of a function 
token is a unique and novel aspect of the present inven- 
tion which permits it to achieve many of its objectives. 

In either, case, the moving portion 44 is a small message 
packet containing information about the function/data 5 
contained in the resident portion 46. For each function, 
there is a master copy of the code defining it in one or 
more of the modules 30. For data, the resident- portion 
is located in the module 30 which produced it. It is the 
moving portion 44 which is broadcast on the global bus 10 
32. As indicated in FIG. 7, the moving portion 44 as 
transmitted to and as received by the modules 30 in- 
cludes such necessary information as a checksum of the 
moving portion 44 for message checking by recipient 
modules 30, a pointer to the master copy of functions or 15 
data, a time tag indicating the time of creation (for 
numerous purposes), etc. The resident portion remains 
in its module 30 and a copy is only transferred to an- 
other module 30 for use if necessary for function perfor- 
mance as will be discussed shortly in greater detail. 

As a further major point of novelty of the present 
invention as mentioned briefly earlier, the global bus 32 
is a serial, multiple access broadcast bus which performs 
a unique round-robin access if a collision is detected and 
avoided. Note that the global bus 32 of the present 25 
invention detects and avoids collisions of data and does 
not destroy two (or more) messages requiring the re- 
transmission thereof as in the prior art contention sys- 
tems mentioned earlier herein. To this end, the read/- 
transmit circuitry portions 36, 38 of the bus 32 require 30 
incorporation of a physical link having passive and 
active states to support “wired-or” arbitration. In the 
embodiment of the present invention being built for 
testing by applicants, the data link protocol will be 
controlled by a VLSI global bus controller (GBC) 60. 35 
The description contained hereinafter contains specific 
reference to field lengths and word sizes with respect to 
that test embodiment; however, those skilled in the art 
will recognize that actual lengths and sizes will depend 
on the equipment employed in the actual building of a 40 
system. Reference should be made to FIGS. 8-10 with 
respect to the discussion which follows immediately 
hereinafter. 

A global bus message 48 consists of a sequence of 
asynchronously transmitted 19-bit words 50. The key 45 
point here is that the message 48 is comprised of a se- 
quence of fixed length portions instead of comprising a 
continuous bit steam delineated by a header block and 
an ending token as in the prior art. Each word 50 in- 
cludes a synchronizing start bit, a 2-bit field which iden- 50 
tifies the word type and a 16-bit information field. A 
complete message consists of an identification word 52, 
two domain destination words 54 (specifying the recipi- 
ents of the message — either limited or “all”), an optional 
time tag word (not shown in the example), and any 55 
number of data words 56, followed by a 16-bit cyclic 
redundancy check (CRC) word 58, which provides a 
checksum of all the preceding information for message 
receipt checking in a manner well known in the art. 

The identification word 52 incorporates a 3 -bit soft- 60 
ware alterable priority field, a 12-bit hardware defined 
module-unique identification field, and a single conten- 
tion bit, which is appended as appropriate. The domain 
destination words 54 can identify logical grouping of 
modules 30 within a cluster. The contention bit is used 65 
to implement a round-robin arbitration mechanism 
among the other modules when a contention has oc- 
curred. 
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The message identification code is used to detect and 
avoid collisions. Each transmitting GBC 60 both trans- 
mits and listens to the bus traffic. It transmits the next 
bit of its message if and only if it did not detect any 
discrepancies between the data transmitted and re- 
ceived for the previous bit. This scheme permits only 
the GBC 60 transmitting the message 48 with the nu- 
merically highest priority and identification to proceed 
with the transmission of its message Any module 30 
which “collides” during the above-described bit-by-bit 
message identification phase, stops transmitting the 
balance of its message and transmits an active high 
“one” during the contention bit time, which is merged 
into the message with which it potentially collided. If 
the contention bit is detected by a transmitting module 
30, that module 30 will abstain from transmitting again 
until all the remaining (lower priority) contending mod- 
ules 30 are successful in acquiring the bus 32, or until the 
20 bus 32 becomes idle. Thus, it can be seen that with the 
unique global bus and method of operation thereof of 
the present invention, not only are actual undesirable 
collisions avoided, but, in addition, all modules 30 are 
guaranteed access to the bus 32 within a known finite 
time interval and lockout of modules 30 as possible in 
the prior art is absolutely avoided. 

The domain destination words 54 are followed by an 
optional 16-bit time tag word indicating the transmit- 
ter’s local broadcast time as obtained from the bus time 
40. The optional time tag word is followed by one or 
more 16-bit data words 56. The trailing CRC word 58 is 
calculated over all transmitted bits (except the start bits 
and the contention bit). 

At the end of the CRC field 58, there is an acknowl- 
edgement bit time frame 62. If any receiving module 30 
has discovered a transmission error it notifies the partic- 
ipating modules 30 during the hardware acknowledge 
bit time 62. In this event, the preceding message 48 
previously received is deleted by all participating mod- 
ules 30, and the original transmitting module 30 resch- 
edules the transmission of the same message. If no non- 
acknowledgement (“NACK”) is indicated within the 
acknowledgment bit time frame 62, successful transmis- 
sion is assumed. This, of course, is completely contrary 
to the prior art approach requiring active acknowledg- 
ment of successful receipt by all recipients addressed by 
a message. This is an important distinction because, 
logically, there is no difference between “no non- 
acknowledgment” and “all acknowledge”. The use of 
passive acknowledgment as employed by the present 
invention, however, is much more efficient. Active 
acknowledgement on a shared medium requires time 
multiplexing of responses and global knowledge of at 
least the number of respondents. The present inven- 
tion’s passive approach eliminates all that. 

Immediately after the reception of a word 50, all 
receiving modules 50 activate their local transmitter. 
Before the transmitting module 30 can transmit the next 
word 50 in the message 48, all participating modules 50 
must indicate that they are ready to receive. They do so 
by inactivating their local transmitters. Normally, mes- 
sages 48 are separated by idle periods which are at least 
equal to the transmission time of one word 50. As 
shown in FIGS. 9 and 10, when a contention is in 
progress, the idle period is shortened to less than one 
word length. 
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THE MESHWORK As indicated in simplified form in 
FIG. 6, each module 30 includes an interface 64 to the 
point-to-point meshwork 34 used to transport data 
between modules 30. The meshwork interface 64 
incorporates bi-directional ports 66 connected through 
a bypass switch 68. As depicted in FIG. 5, discussed 
briefly above, the meshwork 34 is used to interconnect 
the modules 30 in an arbitrary manner as best suited to 
the application; that is, there is no regular 
interconnection scheme as in prior art networks as 
described above. In the above-mentioned testing 
embodiment of applicants, a bank of seven l-to-6 
multiplexers perform the port switching and the data 
link utilizes source-clocked NR Z or Manchester II 
encoded HDLC packets. The meshwork interface 64 is 
shown in greater detail in FIGS. 11-15. 

In applicants testing embodiment, a meshwork inter- 
face 64 consists of a least one VLSI meshwork control- 
ler 70. A meshwork controller 70, in turn, is comprised 20 
of three receivers 72, one transmitter 74, and a switch 
and extension section, generally indicated as 76, used to 
provide intercontroller connections within the mesh- 
work interface 64. Two controllers 70 may be intercon- 
nected to obtain six non-blocking bidirectional mesh- 25 
work ports as depicted in simplified form in FIG. 15. 
Additional ports may be obtained by connecting more 
than two controllers 70; however, in such case, block- 
ing within the interface 64 is possible. 

It should be noted at this point with respect to appli- 30 
cants’ testing embodiment and the results found there- 
from that the serial transmitter and receivers used are 
not ideal. The combination of the transmitter 74, inter- 
connecting wire 82, and the receiver 72 can have an 
unequal rise and fall time. This presents a problem when 35 
transmissions must go through several modules, such as 
in circuit switching. With each successive module, the 
transmitted signal would get narrower, and the data 
would begin to lose its orientation with the clock as 
illustrated in FIG. 13. In the present invention, special 40 
repeaters 84 are used in the meshwork controller 70 to 
help eliminate this problem. A typical repeater 84 is 
shown in FIG. 14. Skewing of the clock is compensated 
fqr by transmitting the inverted version of the received 
clock. The effects of one node, therefore, are negated 45 
by the effects of another. Skewing of the data, with 
respect to the clock, is eliminated by resampling the 
data with the new clock. 

The object of the meshwork interfaces 64 is to estab- 
lish a direct connection between two modules 30 by- 50 
passing, in each case, the modules 30 along the mesh- 
work 34 between the two ends of the connection. An 
end-to-end, circuit-switched connection is established 
initially by the originating module 30 sending an “open 
channel request” packet. This packet is sent by the 55 
originating module 30 to the first intermediate module 
30 and contains the destination address. At the interme- 
diate module 30, the packet is intercepted by one of the 
interface receivers 72. An available forwarding port is 
found and a connection is established between the in- 60 
coming and outgoing links through the interface cross- 
bar switch 78. The “open channel request” packet is 
then forwarded to the next intermediate module 30. If 
an intermediate module 30 determines that the appropri- 
ate output link is busy or has failed, it issues a “negative 65 
acknowledgement” message to the module 30 which 
made the request. The requesting module 30 can then 
either pause before retransmitting or send the message 
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through an alternate path. Those skilled in the art will 
recognize that the foregoing description combines func- 
tions of the meshwork hardware operating in conjunc- 
tion with system software. Routing and acknowledg- 
5 ment decisions, for example, are made in software. 
When the packet reaches its destination, a “circuit 
established acknowledgement” is sent to the originating 
module 30 through the link established. Note that once 
a link is established, it is physically symmetric. Only the 
10 software protocol dictates the order of messages; in 
fact, they can be simultaneous since the link in the pres- 
ent invention is intended to be full duplex. Once the link 
is established, the originating module 30 is free to trans- 
mit its data. Upon receipt of the data, the destination 
15 module 30 checks the data integrity and sends an ac- 
knowledgement to the source module 30. The circuit is 
maintained until one of the linked modules 30 issues a 
“BREAK” signal, which propagates to all intermediate 
modules 30. Within each interface 64 there is a comple- 
tion monitor 80 which watches for the BREAK signal. 
When it is sensed, the connection at the local crossbar 
switch 78 is disconnected, thereby breaking the circuit 
from end to end. 

THE DISTRIBUTED OPERATING SYSTEM 

The operating system is designed to support the 
unique dataflow architecture of the present invention 
wherein, as previously mentioned, moving portions 44 
of tokens 42 are employed to indicate the locations and 
vital characteristics of data and function resident por- 
tions 46 distributed throughout the system and the mod- 
ules 30 thereof. The operating system of the present 
invention is a layered system consisting of a multi-task- 
ing, event-driven kernel, a data manager, and a graph 
execution manager. The operating system resides in 
each module. There is no central controller in the sys- 
tem. As mentioned earlier but worthy of note once 
again, this is a major point of novelty over prior art 
approaches to the problems solved by the present in- 
vention. The use of a fully distributed and autonomous 
operating system makes graceful degradation a virtually 
inherent feature rather than a complex “add-on”. 

The kernel supports standard features such as cre- 
ation and deletion of processes, inter-process communi- 
cation, priority scheduling, time event management, 
and related functions. The kernel is augmented with 
routines for input and output, communications, and 
interrupt handling. The data manager supports sequen- 
tially structured data. It provides a virtual access for 
local processes to all token sets distributed throughout a 
cluster The graph execution manager schedules data- 
flow tasks within its module 30. It evaluates firing rules 
and executes tasks based on the locality of tokens and on 
function priority. Cooperation among the modules 30 is 
accomplished through use of the global bus 32, Global 
token status is broadcast via the global bus 32 and is 
redundantly maintained in each module 30. 

The graph execution manager creates process tem- 
plates within which functions are executed. These tem- 
plates may be given different priority levels under the 
kernel multitasking system. Efficient dataflow graph 
execution requires a method of limiting token accumu- 
lation. Also, the resolution of recursive evaluations, and 
intelligent scheduling, need to be efficiently managed. 
Assigning priorities to graph nodes provides a mecha- 
nism for these purposes. The priority system allows 
other processing to occur in the intervals during which 
tokens may be en route to a temporarily blocked high 
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priority node. Input and output from a dataflow graph 
is handled by system calls from the node function pro- 
grams. 

The operating system executes multiple dataflow 
graphs which may be independent or may exchange 5 
tokens. In addition, dataflow graphs are divided into 
subgraphs to aid software modularity, fault protection, 
and efficient load distribution. Each subgraph is as- 
signed to a domain consisting of one or more of the 
cluster’s modules 30. A particular module 30 may exe- 10 
cute more than one subgraph and may be a member of 
more than one domain. Each global bus message 48 
identifies the domain(s) to which it is directed as previ- 
ously discussed. Only those modules 30 assigned to such 
domains) accept the message 48. Global bus messages 15 
may also be directed to other clusters via the meshwork 
34 connecting the clusters. 

A graph “node” (as opposed to the concept of a com- 
puter being a “node” in token ring networks, and the 
like) may be either a function or another graph. A node 20 
is a self-contained function written in any language. 
Tokens 42 may not be modified by the function. No 
reference is made within the function code to the data- 
flow graph of which it is a part nor to the identity or tag 
of any tokens 42 used or produced. Thus any function 25 
may be used at more than one node in a graph, and even 
in other graphs. A graph which is referenced as a node 
in one or more other graphs has external input and 
output arcs. These arcs connect to arcs in the referenc- 
ing graph(s). A referenced graph is defined as either 30 
queuing or non-recursive. 

Recursive graphs are separately instantiated on each 
reference to them in other graphs. They will behave as 
if a copy of them were included in each graph referenc- 
ing them. A recursive graph may reference itself, al- 35 
though in this case there can be no loops in the graph. 
The recursion is stopped when an instantiation of the 
graph does not output any tokens to itself. 

Queuing graphs are instantiated only once in the 
system. Such graphs have a single input node and a 40 
single output node. There may be an arbitrary graph 
between these nodes. The input node is fireable when 
tokens 42 received from a referencing graph satisfy its 
firing rules. The firing rules are evaluated separately for 
each reference from other graphs. Each firing of the 45 
input node must cause exactly one firing of the output 
node so that the output tokens can be routed back to the 
graph whose tokens caused the input node’s firing. 
Queuing graphs provide an efficient, transparent mech- 
anism for sharing large data structures. 

The code which implements a function at a graph 
node is itself a token. Each node has an input arc from 
which it receives this code as a token 42. This feature 
has several advantages and it is a substantial and major 
point of novelty of the present invention. First, the 55 
migration of code (for functions) and data may be han- 
dled by the same software mechanisms. It may be more 
efficient in some cases to move the code to the data than 
the data to the code. Treating data and code equiva- 
lently allows the design of allocation heuristics which 60 
can exploit this option. Thus, the dynamic allocation of 
tasks to modules 30 for load balancing is handled effi- 
ciently without need of special mechanisms as em- 
ployed and necessary in the prior art. The integrity of 
all transferred tokens 42 is guaranteed by the error 65 
protection features of the meshwork 34. 

Second, checkpointing the state of the system is just a 
matter of capturing its tokens 42, for this also captures 
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the code. An interrupted task is resumed by reloading a 
checkpointed collection of tokens 42. An overlay may 
be produced by replacing code tokens that are no 
longer required with new tokens. 

In long-duration space missions, updates to an execut- 
ing program (known as patches) may be necessary. In a 
system according to the present invention, “patching” 
merely consists of replacing a code token 42. This is 
accomplished without regard to disturbing proper soft- 
ware execution because node firings are atomic It is 
worthy of note at this point, at least in passing, that the 
atomicity of node firings in the present invention (i.e. 
how input tokens are consumed, the node function is 
evaluated, and output tokens are created, all in one. 
non-divisible operation) is accomplished with global 
bus message processing and token management tech- 
niques which are beyond the scope of the present dis- 
cussion. 

A graph node becomes executable when the tokens 
42 at its inputs satisfy the node’s firing rules. High-reso- 
lution dataflow systems have simple firing rules since 
nodes have few inputs and there are simple functions at 
each node. The low-resolution dataflow system of the 
present invention allows more inputs to a node and has 
a versatile set of firing rules. The graphical description 
of a dataflow graph is simplified and execution effi- 
ciency improved by allowing token merging and selec- 
tion to occur at node inputs rather than requiring sepa- 
rate nodes for such operations. Similarly, the output 
tokens 42 from a node may be automatically duplicated 
onto several output arcs without the use of duplication 
nodes. 

In the present invention, the inputs to a node are 
logically arranged into groups. Each of the inputs to a 
group is either a normal consumable arc, a non-consum- 
able arc, or a round-robin collection of arcs. A node is 
fireable when the firing rules for each of its input groups 
are satisfied. After a token on a normal consumable arc 
is assigned to a node firing, it is unavailable for subse- 
quent node firings and will be consumed when the node 
firing completes. After a token on a non-consumable arc 
is assigned to node firing, it remains on the arc and is 
available for subsequent node firings. An input with a 
round-robin collection of arcs feeding it has a token 42 
available whenever there is a token 42 on any of its arcs. 
The arcs are linked in a ring with equal priority given to 
each arc. After a token 42 from one of the arcs is as- 
signed to a node firing, the other arcs will be checked 
for tokens 42 to assign to subsequent node firings. Thus, 
50 an arc that always has tokens 42 on it cannot block 
tokens 42 on other arcs. The individual arcs in a round- 
robin collection are either consumable or non-consuma- 
ble. The simplest input group is a single input (which 
may be a round-robin collection). There must be a token 
42 at the input for its node to be fireable. 

A “merge” group is a set of two or more input arcs 
whose firing rules are satisfied whenever at least one of 
the input arcs has a token 42. These input arcs are given 
unique priorities. If more than one input arc has a token 
42 when the node is fired, only the token 42 from the 
highest priority arc is assigned to the node firing. Other 
tokens 42 remain available for subsequent firings. A 
token 42 is blocked and will not be used as long as there 
are tokens 42 on higher priority arcs. Each of the input 
arcs to a merge group may be consumable, non-consum- 
able or a round-robin collection. 

A “select” group is a set of two or more input arcs 
with a select-control input arc. Tokens 42 on the select- 
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control arc have integer values which select one of the 
group’s input arcs. The group’s firing rules are satisfied 
when there is a token 42 on the select-control arc and on 
the selected arc. When the node is fired, both the select- 
control token 42 and the selected token 42 are assigned 
to the firing. The input and select-control arcs in a 
merge group may be consumable, non-consumable or 
round-robin collections. 

In a system according to the present invention, fault 
protection is entirely a matter of preserving tokens 42. 
As with graceful degradation, this makes fault protec- 
tion easy instead of complex. Tokens 42 may be lost, or 
damaged, or extraneous tokens 42 may be generated. 
The only other matter of concern is the integrity of I/O. 
Successfully dealing with these problems will make the 
system fault tolerant. In applications requiring a modest 
level of fault protection, occasional checkpointing and 
rollback recovery techniques may be adequate. When 
greater levels of protection are needed, “voting” is 
required and can be implemented easily within the oper- 
ating environment of the present invention. Redundant 
subgraphs are scheduled on separate modules in order 
to generate independent results. Redundant graph 
nodes which have identical input data should produce 
identical tokens 42. In applicants’ testing embodiment, 
at selectable points in the redundant subgraphs, the 
operating system will vote on the checkwords of corre- 
sponding tokens 42. Damaged or missing tokens 42 will 
be detected with high probability. 

When noisy inputs from redundant hardware need to 
be voted, the voting is done within a redundantly exe- 
cuted graph node. Each instantiation of the voting node 
receives all of the redundant hardware inputs. Thus, 
each voting node should produce identical output to- 
kens 42, allowing the operating system to do all further 
software voting based on checkwords. 

In a dual redundant system the detection of an error 
is sufficient to indicate a fault in one of the modules 30, 
but does not provide the means for isolation and recov- 
ery. This is adequate for systems having the require- 
ment to halt when an error is detected. Coupled with 
checkpointing, this may be sufficient for functions 
which require correct results, but not in a time critical 
fashion. When timely automatic recovery is required, 
dual redundancy is not sufficient. A triply redundant 
system in which three sets of tokens 42 are exchanged 
provides the means to detect damaged, lost or extrane- 
ous tokens 42. .Through voting, it also makes possible 
the regeneration and repair of token sequences. Other 
functions can continue while recovery processes work 
to isolate and remove the source of error. Even higher 
levels of fault coverage are possible by extending voting 
to four or more redundant inputs. It should also be 
noted that varying levels of coverage can be applied to 
different graph nodes. Overhead for redundancy need 
only be paid for critical functions. This is a novel fault 
tolerance feature of the present invention. 

The hardware features provided in applicants’ testing 
embodiment of the present invention to support fault 
tolerance include a fail-safe identification system on all 
inter-module communications and interface circuits 
designed to restrict hardware faults to a single module 
30 or communication link. The inter-module communi- 
cation protocol guarantees error-free transmission. The 
alteration of routing algorithms to avoid faulty trans- 
mission links is permitted. Also, modules 30 have inde- 
pendent clocks and power supplies. These mechanisms 
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are the basis for the present invention’s approach to 
fault tolerance. 

We claim: 

1. A computer system comprising: 

(a) a plurality of computers, each said computer hav- 
ing first input/output interface means to a com- 
municatiosn network and second input/output in- 
terface means for interfacing a communications 
network, each said second input/output interface 
means including bypass means for bypassing the 
associated said computer with a bit stream of data 
passing through said second input/output interface 
means; 

(b) global communications network means intercon- 
necting respective ones of said first input/output 
interface means for providing respective ones of 
said computers with the ability to broadcast mes- 
sages simultaneously to the remainder of said com- 
puters; and, 

(c) meshwork communications network means inter- 
connecting respective ones of said second input- 
/output interface means for providing respective 
ones of said computers with the ability to establish 
a communications link with another of said com- 
puters through said bypass means of the remainder 
of said computers; and wherein, 

(d) communications between respective ones of said 
computers includes the use of split tokens each 
having a moving first portion which is sent from 
computer to computer and a resident second por- 
tion which is disposed in the memory of at least one 
of said computers and wherein the storage location 
address of said second portion is contained within 
said first portion. 

2. The computer system of claim 1 wherein: 

said split tokens represent both functions to be exe- 
cuted by said computers and data to be employed 
in the execution of said functions. 

3. The computer system of claim 1 wherein: 

said first input/output interface means each includes 
detecting and terminating means for detecting a 
collision between messages being broadcast over 
said global communications network means and for 
terminating the broadcasting of a message in favor 
of a message with superior right to the use of said 
global communications network means whereby 
collisions between messages from a plurality of said 
computers are detected and avoided, said detecting 
a terminating means comprising: 
said messages broadcast by said computers over 
said global communications network means each 
including a leading module identifying portion, 
and 

each said first input/output interface means includ- 
ing means for testing the last bit broadcast con- 
tents of said global communications network 
means as message from its associated said com- 
puter are being broadcast over said global com- 
munications network means on a bit-by-bit basis 
and for terminating the broadcasting of a mes- 
sage when said last bit broadcast contents of said 
global communications network means is not 
identical to the previous bit of the message just 
broadcast by said first input/output interface 
means indicating that another said computer was 
first in time on said global communications net- 
work means. 
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4 . The computer system of claim 3 wherein addition- 
ally: 

said messages broadcast by said computers over said 
global communications network means each in- 
clude a priority code preceding a module identify- 5 
ing portion whereby another said computer which 
was first in time on said global communications 
network means will automatically relinquish said 
global communications network means to the mes- 
sage of another said computer which is of a higher 10 
priority. 

5. The computer system of claim 3 wherein addition- 
ally: 

(a) said messages broadcast by said computers over 
said global communications network means each 15 
include a reserved portion of fixed length and nor- 
mally containing no data immediately following 
said module identifying portion; 

(b) said first input/output interface means at each said 
computer includes first logic means for broadcast- 20 
ing a data bit over said global communications 
network means during said reserved portion when 

it has terminated the sending of a message as a 
result of a detected collision; and, 

(c) said first input/output interface means at each said 25 
computer further includes second logic means for 
testing said global communications network means 
during said reserved portion and for not immedi- 
ately broadcasting a new message following the 
termination of a broadcast message when said data 30 
bit is detected during said reserved portion 
whereby a said computer which has control of said 
global communications network means will relin- 
quish said global communications network means 
for a message of another said computer which has 35 
previously relinquished said global communica- 
tions network means whereby computer lockout 
from access to said global communications . net- 
work means is prevented and equal access to all 
said computers is guaranteed. 

6. The computer system of claim 1 wherein: 

(a) said messages broadcast by said computers over 
said global communications network means each 
include a cyclic redundancy check code for the 
preceding bits and a trailing portion of fixed length 45 
and normally containing no data; 

(b) said first input/output interface means at each said 
computer includes third logic means for testing 
each said message as received by means of said 
cyclic redundancy check code and for broadcast- 50 
ing a data bit over said global communications 
network means during said trailing portion if a said 
received message is in error; and, 

(c) said first input/output interface means at each said 
computer further includes fourth logic means for 55 
testing said global communications network means 
during said trailing portion and for rebroadcasting 
its last broadcast message as not having been re- 
ceived if it senses said data bit during said trailing 
portion. 

7. The computer system of claim 1 wherein: 

(a) said bypass means of said second input/output 

interface means at each said computer includes a 
receiver for receiving inputs to the associated said 
computer, a transmitter for transmitting outputs 65 
from the associated said computer, a plurality of 
inputs and outputs connected to respective ones of 
said inputs and outputs of others of said computers 
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interconnected by said meshwork communications 
network means, and crossbar switch means inter- 
connecting said inputs and said outputs for bypass- 
ing said receiver and said transmitter at the associ- 
ated said computer and for interconnecting one of 
said plurality of inputs to one of said plurality of 
outputs whereby a bypass link between said ones of 
said plurality of inputs and outputs is established. 

8. The computer system of claim 7 wherein: 
said bypass means of said second input/output inter- 
face means at each said computer includes means 
for sensing a unique signal on said meshwork com- 
munications network means indicating that said 
bypass link is to be terminated and for causing said 
crossbar switch means to open said bypass link 
when said unique signal is sensed. 

9. The computer system of claim 7 wherein: 
said bypass means of said second input/output inter- 
face means at each said computer includes repeater 
means for inverting the clock pulses passing 
through said crossbar switch means and for resam- 
pling data passing therethrough with said inverted 
clock pulses whereby narrowing of data pulses and 
loss of synchronization between data passing there- 
through and the clock pulses associated therewith 
is eliminated. 

10. A communications network system with the capa- 
bility of sensing and avoiding message collisions for 
interconnecting a plurality of computers comprising: 

(a) each computer having input/output interface 
means for interfacing to a communications net- 
work; 

(b) global communications network means intercon- 
necting respective ones of said input/output inter- 
face means for providing respective ones of said 
computers with the ability to broadcast messages 
simultaneously to the remainder of said computers; 
and, 

(c) detecting and terminating means included as part 
40 of each said input/output interface means for de- 
tecting a collision between messages being broad- 
cast over said global communications network 
means and for terminating the broadcasting of a 
message in favor of a message with superior right 
to use of said global communications network 
means whereby collisions between simultaneous 
messages from a plurality of the computers are 
detected and avoided; and wherein, 

(d) the messages broadcast by the computers over 
said global communications network means each 
includes a leading module identifying portion; 

(e) said first input/output interface means each in- 
cludes means for testing the last bit broadcast con- 
tents of said global communications network 
means as messages from its associated computer are 
being broadcast serially over said global communi- 
cations network means on a bit-by-bit basis and for 
terminating the broadcasting of a message when 
said last bit broadcast contents of said global com- 

60 munications network means is not identical to the 
previous bit of the message just broadcast by said 
first input/output interface means thus indicating 
that another computer was first in time on said 
global communications network means; and, 

(f) the messages broadcast by the computers over said 
global communications network means each in- 
clude a leading priority code preceding said mod- 
ule identifying portion whereby another computer 
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which was first in time on said global communica- 
tions network means will relinquish said global 
communications network means to the message of 
another computer which is of a higher priority. 

11. The computer system of claim 10 wherein addi- 5 
tionally: 

(a) the messages broadcast by the computers over 
said global communications network means each 
include a fixed length non-data portion immedi- 
ately following said module identifying portion; 10 

(b) said first input/output interface means at each 
computer includes first logic means for broadcast- 
ing a data bit over said global communications 
network means during said fixed length immedi- 
ately following non-data portion when it has termi- 15 
nated the sending of a message as a result of a de- 
tected collision; and, 

(c) said first input/output interface means at each 
computer further includes second logic means for 
testing said global communications network means 20 
during said fixed length immediately following 
non-data portion and for not immediately broad- 
casting a new message following the termination of 

a broadcast message when said data bit is detected 
during said fixed length immediately following 25 
non-data portion whereby a computer which has 
control of said global communications network 
means will relinquish said global communications 
network means to the message of another computer 
which has previously relinquished said global com- 30 
munications network means whereby computer 
lockout from access to said global communications 
network means is prevented and equal access to all 
the computers is guaranteed. 

12. The computer system of claim 10 wherein: 35 

(a) the messages broadcast by the computers over 

said global communications network means each 
include a cyclic redundancy check code for the 
preceding bits and a fixed length trailing non-data 
portion; 40 

(b) said first input/output interface means at each 
computer includes third logic means for testing 
each message as received by means of said cyclic 
redundancy check code and for broadcasting a 
data bit over said global communications network 45 
means during said fixed length trailing non-data 
portion; and, 

(c) said first input/output interface means at each 
computer further includes fourth logic means for 
testing said global communications network means 50 
during said fixed length trailing non-data portion 
and for rebroadcasting its last broadcast message as 
not having been received if it senses said data bit 
during said fixed length trailing non-data portion. 

13. A computer system comprising: 55 

(a) a plurality of computers wherein each said com- 
puter is controlled by a resident copy of a common 
operating system; and, 

(b) communications network means interconnecting 
respective ones of said computers for allowing 60 
respective ones of said computers to broadcast 
messages to all others of said computers simulta- 
neously and for allowing respective ones of said 
computers to transmit and receive data and func- 
tions to and from others of said computers; 65 
wherein, 

(c) the initiation of functions within respective ones 
of the computers is accomplished by the broadcast- 
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ing of function tokens as messages on said commu- 
nications network means; and, 

(d) communication and transfer of both functions and 
data between respective ones of said computers is 
by means of split tokens each having a moving first 
portion which is moved from computer to com- 
puter and a resident second portion which is dis- 
posed in the memory of at least one of said comput- 
ers and wherein the location of said second portion 
is part of said first portion. 

14. The computer system of claim 13 wherein: 

(a) selected ones of said function tokens are duplicate 
tokens which cause the associated function to be 
accomplished on at least two separate ones of said 
computers; and, 

(b) said common operating system at each said com- 
puter includes logic for sensing duplicate tokens, 
for voting on the results produced by the associ- 
ated function at each of said at least two separate 
ones of said computers, and for selecting the said 
results with the highest probability of being correct 
for use as the results of said associated function. 

15. The computer system of claim 13 wherein: 

said broadcasting of said function tokens is initiated 
by the broadcasting of data tokens required for the 
performance of a function as messages on said com- 
munications network means. 

16. The computer system of claim 13 wherein: 

ones of said function tokens have a time of execution 

associated with them and said broadcasting of said 
function tokens is initiated by the associated said 
time of execution being reached. 

17. The computer system of claim 16 wherein: 

said communications network means maintains a 
common system time employed by said computers 
connected thereto for time based activities and the 
time stamping of said tokens. 

18. The computer system of claim 13 wherein: ‘said 
moving firs portion of said split tokens is broadcast over 
said communications network means to all said comput- 
ers simultaneously and said resident second portion is 
transferred over said communications network means 
from one said computer to another. 

19. The computer system of claim 13 wherein: 

(a) each said computer has first input/output interface 
means for interfacing to a communications net- 
work and second input/output interface means for 
interacting to a communications network, each said 
second input/output interface means including 
bypass means for bypassing the associated said 
computer with a bit stream passing through said 
second input/output interface means; and wherein 
said communications network means comprises: 

(b) global communications network means intercon- 
necting respective ones of said first input/output 
interface means for providing respective ones of 
said computers with the ability to broadcast mes- 
sages simultaneously to the remainder of said com- 
puters; and, 

(c) meshwork communications network means inter- 
connecting respective ones of said second input- 
/output interface means for providing respective 
ones of said computers with the ability to establish 
a communications link with another of said com- 
puters through said bypass means of the remainder 
of said computers. 

20. The computer system of claim 19 wherein: 
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said first input/output interface means each includes 
detecting and terminating means for detecting a 
collision between messages being broadcast over 
said global communications network means and for 
terminating the broadcasting of a message in favor 
of a message with superior right to use of said 
global communications network means whereby 
collisions between messages form a plurality of said 
computers are detected and avoided, said detecting 
and terminating means comprising, 
said messages broadcast by said computers over 
said global communications network means each 
including a leading module identifying portion; 
and, 

said first input/output interface means each includ- 
ing means for testing the last bit broadcast con- 
tents of said global communications network 
means as messages from its associated said com- 
puter are being broadcast over said global com- 
munications network means on a bit-by-bit basis 
and for terminating the broadcasting of a mes- 
sage when said last bit broadcast contents of said 
global communications network means is not 
identical to the previous bit indicating that an- 
other said computer was first in time on said 
global communications network means. 

21. The computer system of claim 20 wherein addi- 
tionally: 

said messages broadcast by said computers over said 
global communications network means each in- 
clude a priority code preceding said module identi- 
fying portion whereby another said computer 
which was first in time on said global communica- 
tions network means will automatically relinquish 
said global communications network means to the 
message of another said computer which is of a 
higher priority. 

22. The computer system of claim 20 wherein addi- 
tionally: 

(a) said messages broadcast by said computers over 
said global communications network means each 
include a fixed length non-data portion immedi- 
ately following said module identifying portion; 

(b) said first input/output interface means at each said 
computer includes first logic means for broadcast- 
ing a data bit over said global communications 
network means during said fixed length immedi- 
ately following non-data portion when it has termi- 
nated the sending of a message as a result of a de- 
tected collision; and, 

(c) said first input/output interface means at each said 
computer further includes second logic means for 
testing said global communications network means 
during said fixed length immediately following 
non-data portion and for not immediately broad- 
casting a new message following the termination of 
a broadcast message when said data bit is detected 
during said fixed length immediately following 
non-data portion whereby a said computer which 
has control of said global communications network 
means will relinquish said global communications 
network means for the message of another said 
computer which has previously relinquished said 
global communications network means whereby 
computer lockout from access to said global com- 
munications network means is prevented and equal 
access to all said computers is guaranteed. 

23. The computer system of claim 19 wherein: 
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(a) said messages broadcast by said computers over 
said global communications network means each 
include a cyclic redundancy check code for the 
preceding bits and a fixed length trailing non-data 

5 portion; 

(b) said first input/output interface means at each said 
computer includes third logic means for testing 
each said message as received by means of said 
cyclic redundancy check code and for broadcast- 

10 ing a data bit over said global communications 
network means during said fixed length trailing 
non-data portion as a result of messages received 
with errors; and, 

(c) said first input/output interface means at each said 

15 computer further includes fourth logic means for 

testing said global communications network means 
during said fixed length trailing non-data portion 
and for rebroadcasting its last broadcast message as 
not having been received if it senses said data bit 
20 during said fixed length trailing non-data portion. 

24. The computer system of claim 20 wherein: 

(a) said bypass means of said second input/output 
interface means at each said computer includes a 
receiver for receiving inputs to the associated said 
25 computer, a transmitter for transmitting outputs 
from the associated said computer, a plurality of 
inputs and outputs connected to respective ones of 
said inputs and outputs of others of said computers 
interconnected by said meshwork communications 
30 network means, and crossbar switch means inter- 
connecting said inputs and said outputs for bypass- 
ing said receiver and said transmitter at the associ- 
ated said computer and for interconnecting one of 
said plurality of inputs to one of said plurality of 
35 outputs whereby a bypass link between said ones of 
said plurality of inputs and outputs is established. 

25. The computer system of claim 24 wherein: 

said bypass means of said second input/output inter- 
face means at each said computer includes means 
40 for sensing a unique signal on said meshwork com- 
munications network means indicating that said 
bypass link is to be terminated and for causing said 
crossbar switch means to open said bypass link 
when said unique signal is sensed. 

45 26. The computer, system of claim 24 wherein: 

said bypass means of said second input/output inter- 
face means at each said computer includes repeater 
means for inverting the clock pulses passing 
through said crossbar switch means and for resam- 
50 pling data passing therethrough with said inverted 
clock pulses whereby narrowing of data pulses and 
loss of synchronization between data passing there- 
through and the clock pulses associated therewith 
is eliminated. 

55 27. A computer interconnecting system for intercon- 

necting a plurality of computers for the transfer of data 
systems therebetween comprising: 

(a) each computer having input/output interface 
means for interfacing to a communications net- 

60 work, each said input/output interface means in- 
cluding bypass means for bypassing the associated 
computer with a bit stream passing through said 
input/output interface means; 

(b) meshwork communications network means inter- 
65 connecting respective ones of said input/output 

interface means for providing respective ones of 
the computers with the ability to establish a com- 
munications link with another of the computers 
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through said bypass means of the remainder of the 
computers; 

(c) said bypass means of said input/output interface 
means at each computer including a receiver for 
receiving inputs to the associated computer, a 5 
transmitter for transmitting outputs from the asso- 
ciated computer, a plurality of inputs and outputs 
connected to respective ones of said inputs and 
outputs of others of the computers interconnected 
by said meshwork communications network 1 
means, crossbar switch means interconnecting said 
inputs and said outputs for bypassing said receiver 
and said transmitter at the associated computer and 
for interconnecting one of said plurality of inputs 
to one of said plurality of outputs whereby a bypass 
link between said ones of said plurality of inputs 
and outputs is established; 

(d) said bypass means of said input/output interface 

maens at each computer includnig means for sens- 20 
ing a unique signal on said meshwork communica- 
tions network means indicating that said bypass 
link is to be terminated and for causing said cross- 
bar switch means to open said bypass link when 
said unique signal is sensed; and, 25 

(e) said bypass means of said input/output interface 
means at each computer further including repeater 
means for inverting the clock pulses passing 
through said crossbar switch means and for resam- 
pling data passing therethrough with said inverted 30 
clock whereby narrowing of data pulses and loss of 
synchronization between data passing there- 
through and the clock pulses associated therewith 

is eliminated. ^ 

28. In a computer system comprising a plurality of 5 
computers and a communications network intercon- 
necting respective ones of the computers for allowing 
respective ones of the computers to broadcast messages 
to all the computers simultaneously and for allowing ^ 
respective ones of the computers to transmit and re- 
ceive data and function initiations to and from others of 
the computers, the improvement comprising: 

(a) each computer being controlled by a resident 

copy of a common operating system; and, 45 

(b) the initiation of functions within respective ones 
of the computers being accomplished by the broad- 
casting of function tokens as messages on the com- 
munications network means; and wherein, 
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(c) communication and transfer of both function to- 
kens and data between respective ones of the com- 
puters is by means of split tokens each having a 
moving first portion which is transmitted as a mes- 
sage from computer to computer and a resident 
second portion which is disposed in the memory of 
at least one of said computers and wherein the 
location address of said second portion is contained 
in said first portion. 

29. The improvement to a computer system of claim 
28 wherein: 

(a) selected ones of said function tokens are duplicate 
tokens which cause the associated function to be 
executed in at least two separate ones of the com- 
puters; and, 

(b) said common operating system at each computer 
includes logic for sensing duplicate tokens, for 
voting on the results produced by the associated 
function at each computer, and for selecting for use 
the said results with the highest probability of 
being correct. 

30. The improvement to a computer system of claim 
28 wherein: 

said broadcasting of said function tokens is initiated 
by the broadcasting as messages on the communi- 
cations network of data tokens required for the 
performance of a function. 

31. The improvement to a computer system of claim 
28 wherein: 

selected ones of said function tokens have a time of 
execution associated with them and said broadcast- 
ing of said function tokens is initiated by the associ- 
ated said time of execution being reached. 

32. The improvement to a computer system of claim 
31 wherein: 

the communications network includes means for 
maintaining a common system time employed by 
the computers for time based activities and the time 
stamping of said tokens. 

33. The improvement to a computer system of claim 
28 wherein: 

said moving first portion of said split tokens is broad- 
cast over said communications network means to 
all said computers simultaneously and said resident 
second portion is transferred over the communica- 
tions network only between a computer wherein it 
is resident and a computer requiring it to execute a 
function. 

* * * * * 
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